18-Questionnaire_design

Questionnaire design

Welcome to the questionnaire design guide!

An aim of this course is to develop your ability to translate business problems into actionable research questions and to design an adequate research plan to answer these questions. Therefore, you need to be equiped with knowledge on how to create a survey and properly conduct a research.

Generally, what you can expect from the survey design is similar to what one experiences in a relationship. If you try to take more than you commit, it doesn’t work out. Now on a serious note, if you follow guidelines mentioned here, you will certainly avoid usual traps your fellow collegues were caught in.

In a research process, conducting a survey is a part of (primary) data collection. Before we collect data, we have to make sure that preceding steps are correctly done. However, in the following sections we will focus on the process of designing a questionnaire. Eventually, you will be able to collect relevant data and apply appropriate statistical tests.

Questionnaire design process

A structured questionnaire is a research instrument designed to elicit specific information from a sample of a target population. Usually it is used in a standardized way with fixed-alternative questions (same questions and response options for all respondents).

An objective of a questionnaire is threefold:

• to translate the information need into a set of specific questions that the respondent can and will answer,
• to motivate, and encourage respondents to become involved, to cooperate, and to complete the questionnaire,
• to minimize response error.

In order to meet these objectives, a questionnaire design process suggests the following sequence of steps:

Specification of the information needed

The questionnaire design should be aligned with the research design! In order to do make it aligned, it is necessary to review components of the problem and the approach. In particular, you should review the research questions, hypotheses and characteristics that influence the research design.

If you are interested in the causal effect of one particular (independent) variable on another (dependent) variable, think about an experimental design that might allow you to manipulate this variable. In this case, you particularly have to decide on the following:

• Which variable to manipulate?
  
• Whether to use a between-subjects or within-subjects design?
  
• The cause-effect sequence (the cause must occur before the effect)
  
• The number of experimental conditions
  
• Potential interactions and relationships with other variables (does the effect depend on another variable?)

What you need to be careful about is the effect of reversed causation. The effect refers to the situation where the causal relationship could possible have an opposite direction from what we assumed at the first place. For instance, it is often assumed that an increase in individual income leads to increase in well-being (happiness). However, some researches suggest that this causation could have an opposite direction, i.e. that actually increase in well-being of an individual leads to an increase in income.

Here are some examples of causal research design applications:

• To assess how a product’s country-of-origin impacts attractiveness across different countries.
  
• To analyse the effects of rebranding on customer loyalty.

If you would like to analyze the effects of multiple categorical or continuous (independent) variables on one continuous (dependent) variable, you might use a regression model. When doing this, you particularly have to decide on:

• How to measure the dependent variable (DV). This is particularly important, since you need a variable that is powerful in uncovering variation between subjects (e.g., open-ended questions, such as “How much are you willing to pay for this product” are good candidates). Moreover, you also need to consider the nature of your DV,i.e. whether it is an interval variable, ordinal or categorical variable. The nature of your DV will heavily influence your choice of a correct statistical test.

• How to measure the independent variables (IV) (single-item vs. multi-item scales, categorical vs. continuous). Bear in mind that the nature of the IV, together with DV, affects your choice of a statistical test as well.

• What other variables might cause the effect that you would like to investigate (to prevent omitted variable bias, i.e. variables that are not part of your model but still influence the dependent variable).

• Potential interactions (e.g., is the effect of variable X stronger for group A vs. B?)

Specify the interviewing method

In the next step you should review the type of interviewing method you will use. At this point you need to think in which setting you aim to conduct your survery. For instance, should you do it in a face-to-face setting or rather online. Here you can find some advantages and disadvantages of online surveys:

Additionally, here is the list of the online tools you can use to conduct an online survey (usually for free):

• Qualtrics
• Google form
• Survey monkey
• Free online surverys
• Kwik surveys

Determine the content of questions

In this step you are starting to work on the content of you questions. There are several questions you should ask yourself when writing questions:

• Is the question necessary?
• Will I obtain the needed information?
  
• Are several questions needed instead of one?
  
• What type of data can I collect by asking that question (categorical or continuious)?

In your survey try to avoid asking double-barrelled questions.Those are a single question that attempts to cover two issues. Such questions can be confusing to respondents and result in ambiguous responses. Instead, you might ask multiple questions in order to obtain the inteded information.

Incorrect:
Do you think Nike Town offers better variety and prices than other Nike stores?

Correct:
Do you think Nike Town offers better variety than other Nike stores?
Do you think Nike Town offers better prices than other Nike stores?

Inability and unwillingness to answer

The quality of collected data you highly depends on your ability to address correct participants. Therefore, you need to make sure that your respondents are able to meaningfully answer your questions.

Examples:

• Not every household member might be informed about monthly expenses for groceries purchases if someone else makes these purchases.
  
• Use filter questions that measure familiarity and product use.
  
• Include a “don’t know” option.
  
• If you ask participants for monteray values (e.g. how much are you ready to pay for the XY product?) across several EU, make sure you indicate correct currency (e.g. HRK for Croatia or HUF for Hungary).
  
• Think about how mobile friendly is the layout of your survey (if it is an online survey).
• Good case practices suggest that there should not be more than 2 questions per page (for online surveys displayed on mobile phones).

If you are asking participants to recall certain brands for instance, make sure you use unaided recall question:

Example of unaided recall question:
What brands of soft drinks do you remember being advertised on TV last night?
Example of aided recall question:
Which of these brands were advertised last night on TV?
a) Coca-Cola
b) Pepsi
c) Red Bull
d) Evian
e) Don’t know

If you are asking participants to list something, the good case practice is to minimize the effort required by respondents:

Incorrect:
Please list all the departments from which you purchased merchandise on your most recent shopping trip to department store X.
Correct:
Please check all the departments from which you purchased merchandise on your most recent shopping trip to a department store:
a) Women’s dresses
b) Men’s apparel
c) Children’s apparel
d) Cosmetics
e) Jewelry
f) Other (please specify) ___________

In a case you are asking for information that could be considered sensitive (e.g. money, family life, political beliefs, religion), they should come at the end of the questionnaire. Moreover, it is recommendable to provide response categories rather than asking for specific figures:

Incorrect:
What is your household’s exact annual income?
Correct:
Which one of the following categories best describes your household’s annual gross income?
a) under 25.001 €
b) 25.001€ to 50.000 €
c) 50.001€ to 75.000 €
d) 75.001€ to 100.000 €
e) over 100.000 €

Decide on measurement scales and scaling techniques

Every statistical analysis requires that variables have a specific levels of measurement. Measurement scales you choose for your questions in a survey will affect the answers you get and eventually statistical test you can apply. For instance, it would not make sense to compute an average of genders. An average of a categorical variable does not make much sense. Moreover, if you tried to compute the average of genders defined in numeric values (e.g. male=0, female=1), the output would be interpretable.

Therefore, it is crucial to become familiar with possibilities of each scale before you choose to add another question to your survey. Consequently, chances to obtain data you did not intend to collect and chances that you will not be able to apply tests you intended are significantly lower.

In the following table you can get a quick overview of possibilities per each measurement scale. :

In the table below you can find general procedure for choosing a correct analysis based on the measurement scale of your data and number of variables. It shows statistical analyses we covered during the course and aims to help you choose among them based on the nature of dependent variables on the side, and the nature and the number of your independent variables on the other side:

When it comes to scaling techniques, they are meant to study the relationship between objects. The basic scaling techniques classification is on comparative and non-comparative scales.

The noncomparative scale each object is scaled independently of the other objects. The resulting data is supposed to be measured in an interval and ratio scaled.

Comparative scales (or nonmetric scaling) compare direclty the stimulus object. For example, the respondent might be asked directly about his preference between domestic and foreign beer brands. As a result, the comparative data collected can only be interpreted in relative terms. In the following sections we will walk through both types of comparative scales and briefly introduce them.

Comparative scale: Paired Comparison

• Respondent is presented with two objects and asked to select one according to some criterion.
• The nature of resulting data is ordinal
• Assumption of transitivity (if X > Y and Y > Z, then X > Z) enables the paired comparison data to be converted into a rank order. To do so, you need to indetify the number of times the object is preferred by adding up all the matrices.
• Effective when the number of objects is limited as it requires the direct comparison, and a bigger number of objects makes the comparison becomes unmanagable.
• Example:
  For each pair, please indicate which of the two brands of beer in the pair you prefer.

Comparative scale: Rank Order

• Allow a certain set of brands or products to be simultaneously ranked based upon a specific attribute or characteristic.
• The rank order scaling is a good proxy for to the shopping setting as there are simultanious comparisons of objects.
• The rank order scaling results in the data of ordinal nature.
• Example:
  Rank the various brands of beer in order of preference. Begin by picking out the one brand that you like most and assign it a number 1. Then find the second most preferred brand and assign it a number 2. Continue this procedure until you have ranked all the brands of beer in order of preference. No two brands should received the same rank number.

Comparative scale: Constant sum

• Respondents allocate a constant sum of units (e.g., points, dollars) among a set of stimulus objects with respect to some criterion.
  
• Constant sum is similar to rank order, but it carries specific units.
  
• The resulting data does not just indicate important factors, but also by how much a factor supersedes another one.
  
• Constant sum scaling can be used to observe the comparative significance respondents assigned to various factors of a subject.
  
• Example:
  There are 8 attributes of bottled beers. Please allocate 100 points among the attributes so that your allocation reflects the relative importance you attach to each attribute.

• Basic analysis of constant-sum data involves tabulation of responses and presenting them as either quantities (e.g., “on average, 7 points were allocated to”high alcohol level“), or, as proportions (”On average, 7% of points were allocated to “high alcohol level”).

Non-Comparative Scales: Continuous Rating Scales

• Participants rate the objects by placing a mark at the appropriate position on a line that runs from one extreme of the criterion variable to the other.
  
• One of the advantages of the continuous rating scale is that it is easy to administer.

• Once the ratings are collected, you can splits up the obtained ratings into categories and then assign those depending on the category in which the ratings fall.

Non-Comparative Scales: Itemized Rating Scales

• The respondents are provided with a scale that has a number or brief description associated with each category.
  
• The categories are ordered in terms of scale position, and the respondents are required to select the specified category that best describes the object being rated.
  
• The commonly used itemized rating scales are the Likert, semantic differential and Stapel scales.

Itemized Rating Scales: Likert scale

• Requires respondents to indicate their attitude towards the given object through the degree of agreement or disagreement with each of a series of statements within typically five or seven categories.
  
• Reversed code of some items increases validity.
  
• One limitation is time required to answer a question on a Likert scale. Compared to other itemized scaling techniques, Likert scale is more time consuming as each respondent is required to read every statement given in a questionnaire before assigning a numerical value to it.

In the table below you can find a couple of commonly measured constructs in marketing research such as attitude, importance, purchase intention and similar.

Itemized Rating Scales: Semantic Differential

• Typically, participants rate objects on a number of itemized, seven-point rating scales bounded at each end by one of two bipolar adjectives.

• Semantic differential can measure respondent attitudes towards something (products,concepts, items, people…).

• It helps you find the repondent’s position is on a scale between two bipolar adjectives such as “Sweet-Sour” or “Bright-Dark”. In comparison to Likert scale, which uses generic scales (e.g. extremely dissatisfied to extremely satisfied), semantic differential questions are posed within the context of evaluating attitudes.

• Widely used rating scale in marketing research due to its versatility

When creating a semantical difference question, you should consider the following:

• Number of categories:

• Balanced vs. unbalanced:

• Odd/even number of categories:

• Forced vs. non-forced response

• Verbal description:

Questionnaire structure

The sequnece of questions in a questionnaire could play imporant role. For instance, more sensitive questions (such as demographic-related questions) are usually placed at the end as they can trigger change in respondent’s behavior.

If you plan to conduct an online survey, then you need to think about the respondent’s experience while doing your questionnaire. For instance, spread the content over more short pages and do not have fewer long pages. In online surveys, two questions on one page is a useful rule of thumb. Generally, respondents are reluctant to read and fill out long questionnaire pages. Hence, long pages will lead to a higher dropout rate. In order to reduce dropout rate state how long the survey will approximately take in the introduction of the questionnaire. Take into account that tools like Qualtrics provide the estimated response time in the survey overview.

Consider that the most of people usually use their phones to fill it out. Think about how the questionnaire will appear on a phone screen too. In that regard, think of length of questions especially.

In the end, the questionnaire structure has to be aligned with the research design. For example, if your research design features an experiment, this needs to be reflected in the questionnaire (e.g., you need to assign the respondents randomly to the experimental conditions in case of a between-subjects comparison).

Questionnaire structure for a between-subjects design

In a between-subject design you randomly assign each respondent to different experimental conditions. They would then complete tasks only in the condition to which they are assigned.

For instance, we would like to test the effect of two advertisements on purchase intention. Therefore, one group of (randomly assigned) respondents will be exposed to one advertisement version while the other group (of randomly assigned respondents) will be exposed to another version. After that, both groups of respondents should express their willingness to buy the advertised product. Evenutally, if the dependent variable (e.g. willingness to buy) is measured on interval or ratio scale, then you can use independent t-test to compare group means. The whole experimental design should be organised as following:

Questionnaire structure for a within-subjects design

This type of experimental design involves exposing each respondent to all of the user experimental conditions you’re testing. This way, each respondent will test all of the conditions.

For instance, we would like to test again the effect of two advertisements on purchase intentions, but this time in a within-subject design. First, each respondent will be exposed to the first version of advertisement and right after that asked to rate his/her willingness to buy the advertised product. Subsequently, each participant will be shown another version of advertisement and again rate his/her willingness to purchase the advertised product. Finally, we can compare group means with paired sample t-test (given that data is measured on interval or ratio scale).

Question wording

Generally, question wording should enable each respondent to understand questions and to be able to answer them with reliability. Reliability means that, if a respondent was asked the same question again, he/she would give the same answer again. A number of common problems regarding the question wording have been identified, so we will address the most important ones.

In order to ensure reliability, the issue in terms of who, what, when and where should be defined in each question.
Example: Which brand of shampoo do you use?
Who (the respondent): It is not clear whether this question relates to the individual respondent or the respondent’s total household.
What (the brand of shampoo): It is unclear how the respondent is to answer this question if more than one brand is used.
When (unclear): The time frame is not specified in this question. The respondent could interpret it as meaning the shampoo used this morning, this week, or over the past year.
Where (not specified): At home, at the gym? Where?

A more clearly defined question is:
Which brand or brands of shampoo have you personally used at home during the last month? In the case of more than one brand, please list all the brands that apply.

Use ordinary words. Words should match the vocabulary level of the participants.
Incorrect:
“Do you think the distribution of soft drinks is adequate?”
Correct:
“Do you think soft drinks are easily available when you want to buy them?”

Avoid double negative form. Double negative question forms can confuse respondents, especially when they need to answer with “Agree” or “Disagree”.

Incorrect:
Do you think that it is not uncommon that boys play basketball?
Correct:
In your opinion, is it common that boys play basketball?

Avoid leading questions.Leading questions clue the participant to what the answer should be. Such questions introduce a bias in a particular direction.
Incorrect:
“Is Colgate your favorite toothpaste?”
Correct:
“What is your favorite brand of toothpaste?”

Avoid ambiguous words. Words such as usually, normally, frequently, often, regularly, and other similar words, do not define frequency clearly enough.

Incorrect:
“In a typically month, how often do you go to a movie theater to see a movie?”
a) Never
b) Occasionally
c) Sometimes
d) Often
e) Regularly

Correct:
“In a typically month, how often do you go to a movie theater to see a movie?”
a) Less than once
b) 1 or 2 times
c) 3 or 4 times
d) More than 4 times

Choose adequate order

One of the last steps in a process of designing a questionnaire is choosing adequate order of questions and instructions for respondents.

At the begining, you should provide a short and easy-to-understand introduction to the topic. Use simple language and avoid technical terms (e.g., not many people will know the terms “manufacturer brand” and “store brand”). Additionally, in the introduction you should state how long the survey will approximately take.

The opening questions should be interesting, simple and non-threatening. They are crucial because it is the respondent’s first exposure to the questionnaire and is likely to set the tone for the rest of questions in the questionnaire. If too difficult to understand, or sensitive in some way, respondents are likely to stop answering your questions. Qualifying questions (or screening questions) should serve as the opening questions (if applicable). Their purpose is to identify a potential respondent that is eligable to proceed with the research survey.

After the opening part, you should establish an optimal question flow. General questions should precede the specific questions. Questions on one subject, or one particular aspect of a subject, should be grouped together. It may feel confusing to be asked to return to some subject they thought they already gave their opinions about.

As respondents are moving towards the end of the questionnaire, they are likely to become increasingly indifferent and might give careless answers. Therefore, questions of special importance should ideally be included in the earlier part of the questionnaire.

Finally, you should pay particular attention to provide all prescribed definitions and explanations before you ask a question. This ensures that the questions are undestood in consistent way by every respondent.

Test your questionnaire

Finally, before you distribute the final questionnaire, there are some things to consider. First, you should always pretest your questionnaire before sharing it! Test all aspects of the questionnaire (content, wording, sequence, form & layout, etc.). If possible, use respondents in the pretest that are similar to those who will be included in the actual survey. Ideally, the pretest sample size should be small (in a real scenario this could varyfrom 15 to 30 respondents; for the group project, a lower number will be sufficient). After each significant revision of the questionnaire, conduct another pretest, using a different sample of respondents. Eventually, code and analyze the responses obtained from the pretest so that you make sure that you collected information you intended to collect.

After testing your questionnaire you should be able to determine whether:

• The questions are properly framed
  
• The questions wording triggers any biases
  
• The questions are placed in the optimal order
  
• The questions are understandable
  
• Specifying questions are needed or some need to be eliminated

Questionnaire in Qualtrics

A questionnaire creation in Qualtrics starts with creation of a Qulatrics project. Each project consists of a survey, distribution record, and collection of responses and reports. There are three ways to create a questionnaire.First, you can create a new survey project from scratch. Second, you can create a new questionnaire from a copy of an existing questionnaire. Eventually, you can create from a template in your Survey Library, or from an exported QSF file.

In order to create a completely new questionnaire, you need to do the following:

Go to the Projects page by clicking the Qualtric XM logo or clicking Projects on the top-right.

Create new project by clicking the blue button on the right side.
In the “Create your own” section click on the survey button.

Enter a name for your survey and get started with a survey creation.

If you would like to create a new questionnaire on a basis of an already existing one, then you choose “From a Copy”. Subseqeuntly, you need to indicate the questionnaire you would like to copy. Now you are good to go!

If there is a questionnaire in the Qualtrics Library you would like to use, then you need to choose “From Library”, and indicate one library name in the dropdown menu.

Question Types and Data Analysis

In this chapter we will encounter the nature of data you collect when conducting a survey. It will help you choose a type of a question depending on the nature of data you want to collect and on the type of statistical tests you want to apply.

Multiple choice with a single answer

Multiple Choice with a single answer is a type of closed-ended question that lets respondents select one answer from a defined list of choices.

Type of data you obtain is categorical, and the output comes in the following form:

Q5_1	Q5_2	Q5_3	Q5_4	Q5_4_TEXT
NA	NA	1	NA
1	NA	NA	NA
1	NA	NA	NA
NA	NA	1	NA
NA	NA	NA	1	Vending machines with coffee
1	NA	NA	NA

Data handling

What to do with this data now? First, we need to load it in R and prepare for analysis. The numbers you see in the output R recognizes as integers. In order to conduct statistical modelling and properly visualize our results, we need to convert our data to a factor class.
A factor (or coding variable) represents different groups of data by using numbers (integers). In fact, factors appear as numeric variables, but they hold meaning of labels/names of data groups, i.e. nominal variable. These data groups are represented in a form of ‘levels’.
In our case, our multiple choice question output will contain 4 data groups (‘Grocery Store’, ‘Online shop’, ‘Specialised coffee shop’, ‘other’) after converting it to factor:

# Table
table(qualtrics_long$value)

           Grocery shop             Online shop Specialized coffee shop 
                      6                       6                       2 
                  Other 
                      3 

Visualisation

Second, you might want to visualize your results. In order to do so, the data format needs to be in the appropriate format.Here we proceed with data fromat adaptation from the point where we stopped:

# Converting long format to the visualisation-friendly format
mlc_visualisation <- as.data.frame(table(qualtrics_long$value))

# Naming columns
names(mlc_visualisation) <- c('Place','Count')

# Observing
mlc_visualisation

NA

The simpliest way to visualize data obtained from multiple choice question with a single answer is a bar chart:

## ggplot basic bar chart
labels <- as.character(mlc_visualisation$Place)
barplot(mlc_visualisation$Count, # Column to visualize
        xlab='Places to buy coffee', # X-axis label
        ylab = 'Count(answers)', # Y-axis label
        names.arg = labels,
        main = 'Where do you buy your coffee?') # Title

R package ggplot2 allows you to create visually appealing graphs:

## ggplot2 bar chart
library(ggplot2)
p <- ggplot(data=mlc_visualisation, 
             aes(x=Place, y=Count, fill=Place)) +
             geom_bar(stat='identity') + theme_minimal() # ggplot2 basic barplot
p

Another R library which can help you make amazing interactive charts in a minute is plotly. Here we use a function called ggplotly(), which allows you to turn any ggplot2 chart interactive. Since we have already created a bar chart using ggplot2 and saved it as “p”, we will just turn it into plotly graph:

library(plotly)
ggplotly(p)

An improved version of ggplot2 package is the packaged called ggvis, which is still in developing:

library(ggvis)
ggvis(mlc_visualisation, 
      x = ~Place, 
      y = ~Count, 
      fill=~Place)

• Renderer: SVG | Canvas
• Download

Statistical analysis